fix(parquet/metadata): preserve infinite floating-point statistics - #1007
Conversation
zeroshade
left a comment
zeroshade
left a comment
There was a problem hiding this comment.
Verified: the float statistics seed min/max with MaxFloat32 / -MaxFloat32, which are not identity elements — for a column of all +Inf, min(MaxFloat32, +Inf) stays MaxFloat32, so the writer records a finite bound that never appeared in the data (and symmetrically the max for all -Inf). Seeding with ±Inf is the correct choice, and replacing the sentinel emptiness check with min > max correctly identifies the untouched-default case without false positives: NaN is filtered earlier, and real data always satisfies min <= max. The Float16 path is updated consistently through defaultMinFloat16/defaultMaxFloat16 and cleanStat. LGTM.
The initial CI failure here was an unrelated flake in TestNewBloomFilter, which depends on runtime.GC() running the addCleanup finalizer before its deferred AssertSize; it passed on rerun.
2 participants
Rationale for this change
Floating-point statistics use the largest finite values as their initial min/max bounds. Those are not identity values when a column contains only positive or negative infinity, so the writer records a finite bound that was not present in the data.
What changes are included in this PR?
Are these changes tested?
Yes. The focused statistics tests, including the existing NaN and signed-zero coverage, pass.